Reducing Worst-Case Illumination Estimates for Better Automatic White Balance

Automatic white balancing works quite well on average, but seriously fails some of the time. These failures lead to completely unacceptable images. Can the number, or severity, of these failures be reduced, perhaps at the expense of slightly poorer white balancing on average, with the overall goal being to increase the overall acceptability of a collection of images? Since the main source of error in automatic white balancing arises from misidentifying the overall scene illuminant, a new illuminationestimation algorithm is presented that minimizes the high percentile error of its estimates. The algorithm combines illumination estimates from standard existing algorithms and chromaticity gamut characteristics of the image as features in a feature space. Illuminant chromaticities are quantized into chromaticity bins. Given a test image of a real scene, its feature vector is computed, and for each chromaticity bin, the probability of the illuminant chromaticity falling into a chromaticity bin given the feature vector is estimated. The probability estimation is based on Loftsgaarden-Quesenberry multivariate density function estimation over the feature vectors derived from a set of synthetic training images. Once the probability distribution estimate for a given chromaticity channel is known, the smallest interval that is likely to contain the right answer with a desired probability (i.e., the smallest chromaticity interval whose sum of probabilities is greater or equal to the desired probability) is chosen. The point in the middle of that interval is then reported as the chromaticity of the illuminant. Testing on a dataset of real images shows that the error at the 90 and 98 percentile ranges can be reduced by roughly half, with minimal impact on the mean error. Introduction Illumination estimation is usually the first step in automatic white balancing of digital images. Once the RGB of the illuminant is known, it can be used to adjust the other image RGBs to make a more pleasing image. Many illumination-estimation algorithms have been proposed [2, 13, 15] and they tend to work reasonably well on average, but they often (perhaps, for 1 image out of 50) fail dramatically. The accuracy of illuminant estimates is often measured in terms of the angle in degrees between the estimated versus actual RGB of the illumination. Many methods are able to obtain mean and median angular errors under 4 degrees on the standard image test sets, but at the same time have maximum errors over 25 degrees. Images that are white balanced based on such inaccurate estimates are universally unacceptable. Although the mean and median measures are valuable error measures, users are perhaps more likely to be concerned with the number of failures than with the number of cases that are marginally poorer, say, having an error of 5 degrees instead of 4. We do not evaluate user preference metrics here, but presuppose that the failure cases are important, and propose a way of using the results from existing illumination-estimation algorithms that combines them in a probabilistic way that leads to lower large errors. There are two general approaches to computational color constancy. The goal of the first approach is to create illuminantinvariant descriptors. Examples include [7, 10, 12]. The second approach aims to predict what the image of the scene would be under a canonical illuminant. This is usually accomplished by estimating the RGB of the scene illumination, and then adjusting the image accordingly [2, 13]. This paper focuses on the illumination-estimation step of this second approach. State-of-the-art illumination-estimation algorithms [13] perform well in terms of median error (i.e., the error at the 50 percentile). However, little attention has been given to reducing the errors at the 90 or 99 percentiles. If at least 99% of our images are required to be acceptable then it is the error at the 99 percentile that is important. This paper presents a new method called the Reduced Worst Case algorithm (RWC) for minimizing the errors at any specified percentile. For example, it is possible to tune the algorithm to minimize the 90 or 99 percentile errors. The algorithm combines image features consisting of the output from an existing algorithm such as Edge-based Color Constancy [22] or MaxRGB [9] with a characterization of the image gamut in terms of its minimum rgb (r=R/(R+G+B) etc.) and maximum rgb values [18]. The RWC algorithm is evaluated on the standard SFU database of 321 images [3]. The results show a significant decrease in the high percentile error accompanied by only a modest increase in the median error. The paper is structured as follows: Section 2 describes related work. Section 3 describes the algorithm and features used in more detail. Section 4 presents the results of the accuracy tests for various combinations of features and desired percentile settings. Section 5 is the conclusion. Related Work There are illumination-estimation algorithms [5, 4] that combine multiple clues and/or results from different algorithms using an average or weighted average. Various forms of consensus were explored by Bianco et al. [4]. Machine learning techniques for combining results were explored by Li et al. [16] and Cardei et al. [5]. However, none of these methods attempts to minimize worst-case errors at a given percentile. Perhaps closest to the RWC algorithm proposed here is that of Chakrabarti et al. [6], which uses a multivariate Gaussian probability distribution of features derived from the spatial frequencies of training images of scenes illuminated by a canonical illuminant. As well, there is the algorithm of Schaefer et al. [21] 52 © 2012 Society for Imaging Science and Technology that combines algorithms capable of producing the likelihood of an input image being illuminated by a particular illuminant. The algorithm we propose for combining the features differs from that of Chakrabarti et al. in that a Gaussian (and therefore symmetric) distribution of the feature probabilities is not assumed. Moreover, the algorithm is trained using scenes illuminated by wide set of illuminant spectra and therefore is not limited to vonKries diagonal transforms when constructing the models. Unlike Schaefer’s algorithm, RWC can combine outputs of any algorithm regardless of whether or not they produce likelihood estimates. Neither Chakrabarti’s nor Schaefer’s algorithms focus on minimizing high-percentile errors as proposed in this paper. In terms of minimizing large errors, Cubical Gamut Mapping [18] attempts to minimize the maximal error, but it cannot be tuned to minimize the error for a given percentile. Proposed Reduced Worst Case Algorithm The proposed RWC algorithm estimates each of the three channels of the rgb chromaticity separately. Although any one of the channels in principle can be calculated from the other two, all three are estimated separately in order to improve the stability of the final estimate. Values in each chromaticity channel are quantized into N bins of equal size. Given an input image, the algorithm estimates the probability of the scene illuminant having the chromaticity associated with a particular bin. Once the probability distribution estimate for a given chromaticity channel is known, the smallest interval that is likely to contain the right answer with a desired probability (i.e., the smallest chromaticity interval whose sum of probabilities is greater or equal to the desired probability) is chosen. The point in the middle of that interval is then reported as the chromaticity of the illuminant. Given an accurate probability distribution, the algorithm will not make an error greater than 1⁄2 of the interval in the desired percentage (e.g., 90%) of the images. Figure 1 shows an example of the computed probability distribution estimate for a single channel, an interval covering chromaticity bins with at least 90% chance, and the resulting chromaticity returned by the algorithm. Note that the answer does not have to correspond to the maximum likelihood estimate. Figure 1. An example of the probability estimate of the illuminant’s r chromaticity. The smallest interval covering the desired percentile is shown in grey. The middle of the interval is shown in solid black. The algorithm returns the r chromaticity of 0.3, even though the maximum likelihood estimate for the r chromaticity is 0.2. The algorithm depends on correctly estimating the probability distribution of the illuminant chromaticity as a function of the chromaticity component bins. For each bin ci, a model is created consisting of all feature vectors, Fij, collected from training images Tij, j=1, ....Mi whose actual illuminant corresponds to the chromaticity component bin ci represents. The feature vector Fij contains: • The estimated illumination chromaticity rgb for image Tij provided by each of the underlying algorithms (e.g., from MaxRGB, Greyworld, and Edge-based Color Constancy), • The minimum r, minimum g, and minimum b from Tij, • The maximum r, maximum g, and maximum b from Tij. The minimum rgb and maximum rgb together provide a rough measure of the image’s color gamut. As shown by Forsyth [8], the gamut is a useful feature because the illumination directly affects the gamut of image colors. Given an input image, I, of a scene taken under an unknown illuminant, its feature vector, F, is constructed the same way by concatenating the chromaticity estimates of the underlying algorithms along with the minimum and maximum rgb values. To estimate the probability P(ci|F) that the unknown illuminant’s chromaticity corresponds to bin ci, Bayes rule is used: ( ) ( ) ( ) i i i c P c F P F c P | ~ | (1) The probability P(F|ci) of feature vector F belonging to bin ci is based on Loftsgaarden-Quesenberry multivariate density function estimation [17] for the point in feature space occupied by F relative to all the training feature vectors Fij in the bin ci. The value fi(F) of the probability density function fi at point F is estimated as fi est = k ni ( )−1 ni ⋅ Γ d 2 ( )d 2rπ d 2 (2) where ni is number of training points in the bin ci, k(n) is a nondecreasing sequence of positive integers (smallest integer greater or equal to n is used here), d is the dimensionality of F, r is a radius of the smallest sphere centered around F that covers at least k(ni) training points Fi,j from bin ci, and Γ is the gamma function. The term (Γ(d/2)d)/(2rπ) is an inverse of the volume of a sphere of radius r in a d-dimensional space. Assuming that Pest(F|ci) ~ fi est(F) and removing the constant terms yields